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Abstract. The prevalence of human error in safety-critical occupations remains a major challenge to mission success despite 
increasing automation in control processes. Although various methods have been proposed to prevent incidences of human error, 
none of these have been developed to employ the detection and regulation of Operator Functional State (OFS), or the optimal 
condition of the operator while performing a task, in work environments due to drawbacks such as obtrusiveness and impracticality. 
A video-based system with the ability to infer an individual’s emotional state from facial feature patterning mitigates some of the 
problems associated with other methods of detecting OFS, like obtrusiveness and impracticality in integration with the mission 
environment. This paper explores the utility of facial expression recognition as a technology for inferring OFS by first expounding on 
the intricacies of OFS and the scientific background behind emotion and its relationship with an individual's state. Then, descriptions 
of the feedback loop and the emotion protocols proposed for the facial recognition program are explained. A basic version of the 
facial expression recognition program uses Flaar classifiers and OpenCV libraries to automatically locate key facial landmarks 
during a live video stream. Various methods of creating facial expression recognition software are reviewed to guide future 
extensions of the program. The paper concludes with an examination of the steps necessary in the research of emotion and 
recommendations for the creation of an automatic facial expression recognition program for use in real-time, safety-critical missions. 


1.0 Introduction 

The increasing mechanization of control 
processes in the modern world has 
lessened the gross total workload of the 
operator and has increased the breadth of 
tasks that humans can accomplish. 

However, despite the operator shouldering 
an objectively smaller workload than before 
the advent of automation, human error still 
accounts for the largest percentage of 
actions leading to the prevention of mission 
success. When affected by emotional and 
environmental influences, the operator can 
falter in the execution of critical tasks. In 
order to combat the negative consequences 
of a sub-optimal operator state, it becomes 
necessary to create a system that both 
evaluates the operator and influences him 
or her into embracing an optimal emotional 
state. Operator Functional State (OFS) 
defines the current theory of what 
constitutes an optimal state: “the 
multidimensional pattern of human psycho- 
physiological conditions that mediates 
performance in relation to physiological and 
psychological costs, [and] results from the 
synthesis of operator characteristics, current 
operator conditions, and the operator’s 
interaction with operational 
requirements. ”[13] OFS does not place 
value judgments on a specific aspect of 
state, but rather looks at its holistic 


representation and effect on the operator. 
The regulation of OFS in operators could 
help to spot and prevent any human errors 
from occurring. 

Various techniques have been proposed 
and utilized to measure OFS in real-time, 
such as electroencephalography (EEG), eye 
activity, and core temperature 
measurements. [13] The implementation of 
these techniques allows for the least noisy 
sources of data; however, each method only 
measures a single aspect that affects OFS, 
and the implementation of the multiple 
apparatuses measuring these different 
methods onto a single person would cause 
considerable discomfort and diminishing 
returns in respect to the amount of useful 
data acquired. The most important 
component of assessing OFS becomes 
finding a measurement that can take 
multiple factors into account while 
extrapolating from a single source of 
information. This source of information can 
be found in the human face. 

Humans have residual information about 
their current activities and emotions stored 
in their face for a variable length of time, 
and can be charted through tracing muscle 
movements. Researchers have employed 
electromyography (EMG) in order to 
measure muscle activity in the face because 
of its reliability in detecting facial muscle 
movement. [9] However, the method uses 
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electrodes attached to the face, which has 
several technical and practical 
drawbacks. [9] In the setting of a flight deck, 
for example, the inconvenience of attached 
electrodes overshadows the benefits. A 
potential non-obtrusive system comes in the 
form of monitoring an operator through live- 
video streams, and by using facial 
expression as a measure of state. This 
paper overviews the science behind facial 
expressions, their link to emotions, and 
describes stress, fatigue, and complacency: 
three psycho-physiological states that 
promote sub-optimal OFS. Next follows a 
description of the feedback loop video 
system, along with the suggested protocols 
and models for the regulation of an 
operator’s state in the case of stress, 
fatigue, and complacency. Then, a 
description of the basic program created by 
the author, and an explanation of the 
computer algorithms and statistical models 
available to create a facial expression 
recognition program shows the extent of the 
possibilities in computer vision. 

2.0 Emotion and OFS 

Darwin’s work, The Expressions of 
Emotion in Man and Animals, postulates 
that human emotions, and subsequently the 
physical generation of emotion through 
facial expressions, are “evolved and 
adaptive.” Darwin proposes that human 
emotion and its expression could not be the 
product of societal constructs, but represent 
the culmination of millions of years of 
evolution selected to benefit the species. [8] 
As social creatures even before the advent 
of human speech, ancestral humans could 
communicate their emotions and intentions 
to others through their facial expressions. [8] 
Despite the formation of language, humans 
keep their ability to show their emotional 
state through the face. Ekman and his 
colleagues later revived interest in Darwin’s 
theory of universality of emotion by 
continuing cross-cultural studies and noting 
the similarity of emotional expressions 
throughout the world. Ekman and his 
colleagues agreed with Darwin and proclaim 


that facial expressions, and the emotions 
displayed by each facial expression, are 
universal. [10] In order to create a scientific 
and methodological manner of determining 
a facial expression, Ekman also created the 
Facial Action Coding System (FACS), which 
uses facial muscle movements, defined as 
Action Units (AUs), to discern emotion. 
Vetted by the scientific community, FACS 
presents a foundation for all measures of 
facial expression and continues to be a 
measuring scale for human emotion. FACS 
allows the scientific examination of facial 
muscles in facial expression analysis, which 
can be used to measure OFS after 
examining the effects of emotion. 

Emotion has a continual effect on an 
operator’s perception of a task and on his or 
her OFS. Studies have shown that 
emotional arousal has an effect on the 
general reactivity and decision-making 
abilities of an individual, despite the 
emotion’s valence. [16] The impact of 
emotion on cognitive tasks elevates it to an 
aspect of state to be examined. The face 
allows a unique combination of both 
physiological and psychological changes in 
the body. For example, the perception of 
happiness in an individual connotes an 
emotional recognition fueled by the 
physiological conformation of different 
muscles in the face. The universality of 
facial expressions also validates the 
creation of a facial recognition system 
focused on the determination of sub-optimal 
OFS since it would not have to be calibrated 
for each individual operator. The different 
conformation of muscles can be measured 
through the FACS, which already decoded 
the different potential muscle movements in 
the face. Although different states of sub- 
optimal OFS exist, the following three 
incorporate a large spectrum of the 
challenges faced by operators and will be 
concentrated. 

2.1 Stress 

The Lazarus definition of stress, “the 
process of appraising [negative or 
cumbersome] events, of assessing one’s 
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potential to control or cope with the event, 
and continuing reappraisal as new 
information becomes available,” defines 
stress as a process for dealing with harmful, 
emotional events rather than a strict 
emotional state in itself.[14] Stress does not 
constitute an emotion, but rather combines 
an amalgam of emotions in reaction to a 
situation. Stress can arise from a time- 
restricted and work-intensive environment, 
like those found in safety-critical missions, 
and has both psychological and 
physiological effects on the body that leads 
to a sub-optimal OFS. 

In terms of the psychological effects of 
stress, the brain attempts to mount multiple 
defensive measures, such as a reflexive 
completion of tasks without cognitive input, 
in order to reduce stress. [14] However, 
following the failure or lack of initialization of 
these defensive measures, the mind suffers 
from a reduced ability to process the 
environment due to the high workload. 

Subsequently, stress affects an 
individual’s physical state as well. Selye’s 
General Adaptation Syndrome (GAS) 
represents the process in which stress 
manifests in the physiology of an organism. 
GAS defines the “progressive responses to 
prolonged stress in which an organism 
mobilizes for action and compensates for 
stress,” and its component, Alarm, 
Resistance, and Exhaustion (ARE), shown 
in Fig. 1 , delineates the stages of the 
response. [14] The GAS shows the general 
wearing down of an organism in the face of 
stress. Ultimately, when stress fails to be 
mitigated, the organism reaches exhaustion 
and becomes unable to respond 



Figure 1. Alarm, Resistance, 
Exhaustion (ARE). [14] Figure shows 
the three stages of Selye’s General 
Adaptation Syndrome (GAS). 


appropriately to the environment. The 
culmination of stress, psychologically and 
physiologically, creates an error-prone 
environment for the operator due to his or 
her inability to cope with high-load tasks. 

2.2 Fatigue 

Fatigue colloquially describes “feelings 
of fatigue, exhaustion, and to problems of 
attention and motivation. ”[5] However, the 
amorphous boundaries of this definition of 
fatigue encompass a multitude of other 
emotional states like drowsiness and 
intense hunger. In the context of work- 
related fatigue, fatigue defines a “normal, 
healthy response to an oncoming depletion 
of resources, caused by the execution of 
physical and mental tasks,” which highlights 
the fact that humans are not meant to 
concentrate on a task for an extended 
period time without regard for their overall 
state of being. [5] Work-related fatigue does 
not include the effects of outside influence 
like lack of sleep, although these factors can 
be confounded and mistaken as fatigue. 
Fatigue occurs in a work environment due 
to extended interaction with a task without 
any breaks or change of requirements in the 
task. Fatigue affects the operator’s psycho- 
physiological state by inducing a state of 
tiredness usually followed by a decrease in 
cognitive function. 

The effects of fatigue lead to the 
operator being unable to completely 
process his or her environment. The 
monotony of the task causes the operator to 
revert to more energy-saving strategies. For 
example, in response to the decrease in 
awareness, the operator either continues 
the task at his or her current pace and 
makes mistakes, or slows down the process 
in an attempt to reduce the mistakes. [5] 
Essentially, the operator has to take the 
time to evaluate different work strategies to 
combat the side effects of fatigue. 
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2.3 Complacency 

Complacency, in the terms of safety- 
critical missions, defines the tendency for 
operators to blindly trust automation without 
monitoring, and often renders the operator 
unable to quickly respond to any 
malfunctions. [12] Complacency also results 
when the operator works in a highly reliable 
and automated environment, and the 
operator’s sole task is to stay vigilant in 
case of occasional error.[15] Although 
automation has increased the capability of 
an operator in a multitude of environments, 
it has also caused an unbalanced mental 
workload and reduced situational 
awareness in operators.[12][15] 
Complacency differs from boredom due to 
the implicit trust in the automation; 
complacency lulls the operator into a false 
sense of security due to trust. While an 
operator may be bored because of the 
absence of a task, a complacent operator 
can also be bored due to the trust given to 
the automation. A complacent operator 
appropriates a small cognitive investment in 
his or her environment, and can lead to 
overlooking any number of mistakes in the 
automation. 

Complacency does not necessary 
increase the number of errors committed by 
the operator, but rather causes the operator 
to ignore potentially dangerous situations 
due to a lack of cognitive awareness. 
Complacency leads to a reduced reaction 
time if any errors occur, or an inability to 
spot any errors at all. If an error occurs 
while the operator is complacent, the 
operator could be sent into a panic while 
attempting to discover and fix the problem 
occurring in the task simultaneously. 

3.0 Video Systems and Facial 
Expression Recognition 

3.1 The Feedback Loop 

The use of a feedback loop allows 
information gathered from the user to be 
analyzed in real-time and sent back to 
inform him or her to modify or continue his 


or her current actions. The proposed 
feedback loop, as shown in Fig. 2, notifies 
the user about their state based on the data 
analyzed from the facial expression 



Figure 2. The Feedback Loop. The 
feedback loop utilized in the potential 
facial recognition system in order to aid 
pilots and other critical-safety task 

recognition program. The program then 
sends the product of the data analysis back 
to the user in the form of useful information 
regarding their current state. For example, 
when the subject displays a sub-optimal 
state of awareness, the feedback loops 
allow this information to be sent back to the 
user. Users can then modify their state 
based on their newly acquired knowledge, 
or, if they lack the ability due to extreme 
conditions, allow other protocols to be put 
into place. The feedback sent to the 
operator would depend on the emotion he 
or she displayed, and would be designed to 
combat that specific sub-optimal state. The 
different protocols that would be assigned to 
the three main sub-optimal OFS, and the 
make-up of the physical systems in the 
feedback loop, are explained further. 

3.2 Suggested Protocols for Sub- 
Optimal States 

3.2.1 Stress Protocol Description 

The weight of stress on an operator 
leads to a variety of negative emotional 
responses such as fear, anger, disgust, 
distress, and helplessness. [4][1 1] Although 
a distinct stress progression does not 
project itself directly onto the face, the 
stress can be traced through the shifts 
between negative emotions on the face in 
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response to high-load work. The program 
would monitor the operator for any 
sequence of negative facial expressions 
over a period of time. Current research 
shows that the detection of stress through 
live-video streams is both possible and 
improving and could be an integral part of 
assessing OFS.[4] 

The program would constantly monitor 
for an emotional reaction to stressors, like a 
sudden change in the environment, and 
note the duration of the emotional response. 
If stress episodes are infrequent and short 
in duration, the program will not report those 
results to the operator. However, once the 
length or frequency of the episodes 
increase, then the program would report the 
findings to the operator through voice- 
command or through a notice in one of the 
display monitors. The program then prompts 
the operator to commence a relaxation 
technique, such as deep breathing or 
voluntary and directed muscle contraction. 
The implementation of the stress protocol 
hopes to reduce stress as it begins, rather 
then allowing the operator to reach 
maximum stress levels at any point. 
However, if the levels of stress are deemed 
to be overwhelming the operator, then the 
program will suggest replacing the operator 
if possible. 

3.2.2 Fatigue Protocol Description 

Although the measurement of fatigue 
should not include the symptoms of sleep 
loss or an out-of-sync circadian rhythm, the 
outward manifestation of these states are 
similar.[5] Therefore, for the purposes of the 
fatigue protocol, all states that manifest 
similarly to fatigue will be considered as 
such. However, while fatigue can be 
alleviated through the implementation of 
short breaks or modifying the task, the 
curing of symptoms caused by sleep loss 
require more intensive recovery. In light of 
the similarities between the states, the 
program will also pay attention to the 
duration that the emotion outwardly affects 
the operator. 


Although the current literature does not 
describe the facial expressions associated 
with fatigue, the proposed program looks for 
characteristics like heavy-lidded eyes 
combined with a neutral expression and 
possibly the inability to detect the operator’s 
face for increments of time due to head- 
nodding or complete eye closure. The 
program monitors the operator for their level 
of fatigue throughout the task. The goal of 
the program is to squash any signs of 
fatigue before they affect the operator in a 
fundamental way and reduce work 
efficiency. If the level of fatigue passes a 
level to be determined, the program would 
inform the operator and instruct them to 
take a break, focus on another task, or 
instruct the task-manager to appropriate 
another task to the operator. 

3.2.3 Complacency Protocol 
Description 

Complacency generally manifests itself 
in ways similar to boredom. In terms of 
OFS, complacency usually occurs from the 
under-load experienced due to automation. 
Research shows that the most effective way 
to place the operator back into experiencing 
the appropriate level of load comes in the 
form of adaptive automation. [12] Adaptive 
automation refers to the ability to reallocate 
the level of automation from the operator to 
the environment depending on the 
operator’s state. [12] 

The current literature does not describe 
the facial expressions associated with 
complacency, however, for the purposes of 
the program, complacency will be discerned 
through the formation of a neutral face, 
combined with the appearance of cues like 
constant pupil movement away from 
potential locations of display monitors. If the 
operator appears complacent, then the 
program will alert the adaptive automation 
system to return some function back to the 
operator. The transfer of work back to the 
operator will allow them to regain focus and 
invest some cognitive awareness into his or 
her environment. 
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3.3 The Physical Components 

3.3.1 Webcam 

Since their introduction, webcams have 
improved in both their popularity and their 
capabilities; however, they still suffer from a 
multitude of technological shortcomings, 
such as the maximum number of frames per 
second and megapixels. The webcam also 
experiences some practical limitations, like 
the inability to function well in poorly lit 
environments. [9] However, despite these 
setbacks, webcams allow an inexpensive 
alternative to expensive video equipment; 
webcams are built to stream video live onto 
another source, while more traditional video 
equipment solely stores video for later use. 
Webcams also have the advantage of size, 
and the introduction of a webcam into the 
work environment will be almost 
unnoticeable due to their small size. 

3.3.2 Thermal Camera 

Preliminary research shows that thermal 
cameras present another non-obtrusive way 
to detect facial muscle movement. [9] By 
analyzing the muscles delineated in the 
FACS, the researchers found that the 
production of AUs creates an increase in 
the temperature in certain areas of the face 
while decreasing other areas, and shows a 
promising method of evaluating facial 
muscle movement.[9] Programs that rely on 
webcams suffer from the camera’s inability 
to register images correctly when the 
subjects are under poor lighting. Thermal 
cameras, however, do not require any light 
at all; the images form through the analysis 
of heat emanating from a surface. A 
drawback of thermal cameras comes from 
the camera’s inability to see through other 
objects. For example, if an operator had to 
wear goggles, the thermal camera would 
only reflect the heat of the goggles rather 
than any of the eye muscles due to the 
goggles occluding the eyes. The occlusion 
occurs for all objects, regardless of 
transparency, so any accessories that cover 


the face will have to either be removed or 
redesigned to prevent facial occlusion. 

The combination of both standard 
webcams and thermal cameras in the 
system would allow for both the continual 
monitoring of the operator despite lighting 
conditions and the ability to corroborate 
facial muscle movements between the 
webcam and the thermal camera when the 
operator sits under good lighting. The 
utilization of both methods would help to 
cover some of the drawbacks of either 
method and create a more balanced 
system. 

4.0 Computer Algorithms for 
Deciphering Facial Expressions 

The current iteration of the automatic 
facial recognition program, created in Visual 
C++, utilizes Open Computer Vision 
(OpenCV) libraries and Haar cascades, and 
tracks a human face in a live video-feed. 
After identifying a human face, the program 
then subsequently identifies some key facial 
landmarks: the eyes, pupils, nose, mouth, 
and the corners of the mouth. Some of the 
limitations of the program, both intentional 
and unintentional, are as follows: the 
program currently only tracks one individual 
at a time, has a slight delay due to the 
application of the facial landmark 
identification boxes onto a live-stream 
video, does not include thermal camera 
integration, and can only identify facial 
features deviating 30 degrees either to the 
left or right from facing the camera directly. 

The program also lacks a key 
component: facial expression recognition. 
The current state of computer vision allows 
for the utilization of a multitude of algorithms 
to create a program that automatically 
recognizes facial expression. The most 
prominent algorithms seen in recent years 
are Haar cascades, Active Shape Models, 
Active Appearance Models, and the 
Piecewise Bezier Volume Deformation 
Model. 
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4.1 Haar Cascades 

Haar cascades, described by Viola and 
Jones [2001], describe a method to quickly 
identify images using feature-detection. 

Haar cascades rely on Haar-like features 
that detect the change in contrast values 
between adjacent rectangular groups of 
pixels, rather than the intensity values of a 
pixel itself.[19] Although the use of Haar-like 
features reduces computing time drastically, 
comparing every feature in an image in 
order to find an object creates unnecessary 
work. [19] The cascading of classifiers 
allows the further reduction of necessary 
computing power. Cascading refers to the 
utilization of a degenerate decision tree that 
processes images through a series of 
comparison tests, and rejecting the image if 
it fails any one of the tests. [18] For example, 
if the part of the image being compared 
matches the feature in the first sub-window 
but does not match the second, then the 
cascade rejects that portion of the image 
and moves to the next feature. The 
degenerate decision tree allows for the 
comparison of all pixels without an 
unnecessarily complete classifier 
comparison, and results in the rejection of 
most negative images and the approval of 
most positive images. 

In order for a Haar classifier cascade to 
identify a particular object, the algorithm has 
to be trained. The training of the algorithm 
requires a large library of both positive and 
negative images. The positive images 
contain the object that the user wants the 
algorithm to detect, while the negative 
images consist of random images without 
the object. [18] The robustness of the 
algorithm has a positive relationship with the 
number of images used for training; 
however, this positive relationship occurs 
due to the larger probability of variety in 
larger samples. In the case of facial feature 
detection, the sample images would have to 
span across ages, genders, and races. 

For facial expression detection, 
however, Haar cascades create some 
difficulty. Facial expressions, while 
universal, can also be repressed or faked to 


some degree. While noses and eyes vary 
solely in shape and size, facial expressions 
vary in the totality of their expressions, from 
the arch of the brow to the movement of the 
mouth, and the differences between two 
different emotions could be too subtle for 
the Haar cascade to differentiate between. 
Instead of whole-face facial expression 
recognition, it could be possible to track 
more emotive landmarks of the face, like the 
eyebrows, and then create a program that 
would note the locations of facial features in 
relation to the proportions of the face, and 
then process that information for emotional 
detection. However, that process would be 
tedious and probably inaccurate for the 
subtleties of emotional expression. 

4.2 Active Shape Models (ASM) 

Cootes and Taylor’s [1995] ASMs 
illustrate a trained statistical modeling 
algorithm that utilizes Point Distribution 
Models (PDMs) to rapidly locate the desired 
structure. [3] PDMs describe a training set of 
images dotted with landmark points. The 
training sets have multiple instances of the 
same object, but with some natural 
variation. The mean position of the 
landmark points in relation to the object, and 
then the points that show how each object 
in the training set deviates from the 
calculated mean, are determined. [3] After 
the establishment of the PDMs, the ASM 
performs an iterative search to locate the 
shapes of the structures in different 
images. [3] The ASMs result in the ability to 
fit the model to the image after a finite 
number of iterations. 

The human face, although variable, 
generally follows a model format with the 
same number of facial features. The general 
shapes of the facial features are also the 
same, although the size of these features in 
relation to each other, and in general, can 
change. ASMs then provide a non-rigid 
modeling approach to identify and conform 
to different facial expressions. The 
measured facial expressions would be 
trained by different sets of PDMs and then 
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the conformation of certain ASMs would 
denote that expression. 

However, the use of ASMs in order to 
describe human emotion runs into similar 
problems as Haars cascades - although 
ASMs require less training, they only 
identify models defined in their training sets. 
Therefore, a PDM would have to be created 
for each emotion. Even though ASMs are 
more forgiving than Haars cascades when 
conforming to images, the complexities of 
an emotion would have too much deviation 
from the mean. For example, the 
manifestation of a smile can range from the 
open mouth smile to the Duchenne smile. 
Each variation of the smile still constitutes a 
smile; however, the difference between a 
closed-mouth smile and an open-mouth 
smile, combined with the appearance of 
teeth, only constitute a portion of the 
variation. 

4.3 Active Appearance Models (AAM) 

AAM, also created by Cootes and Taylor 
[2001], shows a statistical modeling method 
similar to ASM, but diverges from tracking a 
specific object and moves toward matching 
a class of deformable objects. [2] AAM 
works similarly to ASM by requiring a 
training set marked with points. An image in 
the training set, after being marked, has its 
points aligned; the subsequent wrapping of 
the images around the mean of the training 
set then creates a “shape-free patch,” which 
lacks any texture. [2] The process is 
repeated for each image in the set, training 
the model to conform to a variety of shapes 
of a similar class. The “shape-free patch” is 
raster scanned into a texture vector and 
then applied with Eigen-analysis to build a 
texture model. [2] Lastly, the correlations 
between shape and texture generate a 
combination of the shape and the texture to 
create a combined appearance model. [2] 

The ability to follow deformable objects, 
such as a face with any expression, 
bestows the ability to identify the expression 
without specific training, allowing AAMs to 
surpass AAMs in facial expression 
recognition. AAM works slightly slower than 


ASM, but the added robustness and the 
reduction in different training sets for 
different emotions overrules any 
advantages of ASM in terms of identifying 
facial expression. [2] AAM, however, does 
require a large set of images for training, 
and the availability of large facial databases 
are limited. [17] Until more high-quality 
databases are established, AAMs will not be 
able to attain the robustness capable of the 
model. 

4.4 Piecewise Bezier Volume 
Deformation Model (PBVDM) 

The PBVDM utilizes the Connected 
Linear Deformation Method (CDLM), from 
the work of Tao and Huang [2003] and 
defines a model consisting of multiple 
deformable 2D patches whose movements 
are determined by a set of different vibration 
modes. [7] Vibration modes refer to the 
different displacement vectors that 
determine the deformation of a particular 
patch. These patches are connected 
together with hinges that work to maintain 
the spatial relationships between the 
patches and prevent any drifting. CDLM can 
therefore be used to track and model the 
movements of facial muscles, which would 
be represented as different vibrations of the 
patches. [7] 

The PBVDM creates Bezier volumes 
across specified areas of the face, with a 
section of the surface model, or facial mesh, 
in between the top and bottom volume 
layers. The movement of the Bezier 
volumes deforms the surface model, and in 
the case of facial mesh, shows the 
movement of facial muscles in that region. 
PBVDM can utilize the AUs of the FACS to 
define the available facial movements. 
Therefore, the PBVDM allows a more 
specific analysis of desired facial muscle 
movements in real-time. [6] 

The use of PBVDM in facial expression 
recognition allows a more scientific 
approach. The other models require training 
sets to tell the model what constitutes an 
emotion, and then the models can only 
identify those trained emotions. PBVDM, on 



NASA USRP - Internship Final Report 


the other hand, allows an analysis of muscle 
movements through the use of individual 
volume patches without training sets. 
PBVDM can be made to be analogous with 
a FACS analysis of a face. Current PBVDM 
facial meshes do not cover the complexity 
of facial muscle movement covered by 
FACS, or the ability to decipher emotions as 
effectively as a human can, but the 
accuracy of the algorithms is improving. 

5.0 Discussion 

Currently, the research surrounding the 
emotional states of boredom and fatigue are 
limited in both their breadth and depth. More 
research into the states and facial 
expressions associated with fatigue and 
complacency would be helpful in creating 
algorithms to define those emotional states. 
A potential method of determining the facial 
expressions associated with fatigue and 
complacency would be to conduct a study 
where individuals would be exposed to 
situations that lead to those emotional 
states, and then facial expression would be 
analyzed to search for commonalities, 
similar to the current studies on stress. [4] 

The current iteration of the program 
developed by the author allows for the 
detection of some facial landmarks, which 
represents the first step in automatic facial 
expression recognition. The program could 
be extended even further to recognize other 
landmarks like the eyebrows. In the future, 
the actual creation of an automatic facial 
expression recognition program for use in 
human factor’s experiments will help in the 
tracking of sub-optimal states of awareness 
in operators. The facial expression 
recognition program should be created 
through the use of either AAMs or PBVDM 
due to their robustness and ability to reliably 
deform to the human face. Unless more 
facial databases come into fruition, the use 
of PBVDM would be optimal due to its ability 
to deform to the human face without the 
need for high quality training sets. 

After the creation of a proper facial 
expression recognition program, a complete 
study to evaluate the emotion protocols and 


their effect on the frequency of errors during 
flight would help to determine the efficacy of 
the system. 

6.0 Conclusion 

The study of OFS requires the 
consideration of a large number of variables 
in order to find the optimal OFS. However, 
the use of automatic facial expression 
recognition could help to reduce the number 
of accidents due to human error by helping 
to reduce sub-optimal states of OFS before 
they overtake the operator. The use of video 
systems in lieu of more obtrusive methods 
of determining state allows the practical 
implementation of state-measuring devices 
into the operator’s environment. The 
implementation of video-systems in a work 
environment would help to increase the 
safety experienced by both operators and 
their charges, and lead to a safer world. 
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